Relevance Auras: Macro Patterns and Micro Scatter

Empirical analysis of relevance assessments can illuminate how differ groups of readers perceive the relationship between bibliographic records and index terms. Reported here is an experiment that harvested relevance assessments from two groups, engineers and librarians. These groups assessed the relevance relationships between bibliographic records and index terms for three literatures: engineering, psychology and education. Not only was the indexer-selected term (the topically relevant term) assessed, but also broader, narrower and related terms. Figures 1-8 show these terms arranged as two-dimensional term domains. Positive relevance assessments plotted across the two-dimensional term domains revealed regular patterns, here called "relevance auras." A relevance aura is a penumbra of positive relevance, emanating from a bibliographic records across a term domain of broader, narrower and related index terms. This experiment attempted to compare the relevance auras produced by engineers and librarians at both a macro and micro level of aggregation. This experiment showed that relevance auras appear in data aggregating reader groups and literatures. Mico analyses of individual records, however, showed that relevance auras were ragged or did not develop. Agreement in relevance assessment appears on the individual term basis and often independently of the formation of any relevance aura. Relevance Assessment Mizzaro's (1997) review of the history of relevance studies reveals both the centrality of the concept, as well as how little progress has been made in its definition and quantification. Green considers the use of the term relevance to be in "disarray" (1995, p. 647), citing problems of both theoretical definition and operational measurement. This experiment premises that assessing the relevance of an index term for a bibliographic record is an act of reading that occurs within a cultural context. Transactional reading theory suggests that no two readers will ever generate exactly the same meaning from a text because they bring different backgrounds to the act of interpretation (Straw, 1990). Hjorland and Albrechtsen (1995) stress the effect of language communities on the interpretation of text in their domain-analytic paradigm. This experiment also assumes that harvesting and comparing relevance assessments reveals, in part, the fundamental phenomenon of verbal scatter. In short, people tend to disagree about the names of things (Furnas, et al., 1983). Therefore it is not surprising to find online searchers disagreeing with the indexer's choice of terms. Farrow (1991) reminds us that there is no psychological basis for indexing. Thus, sources of variation in empirical experiments comparing relevance assessments include …